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ABSTRACT 



Between 1990 and 1997, 159 schools in New York were 



classified as Schools under Registration Review (SURR) , but by 1997 54 of 
these schools had improved enough to no longer be classified as SURR. Seven 
of the schools had reorganized or closed, and 98, 94 of which were in New 
York City, were still classified as SURR. An audit was conducted to determine 
whether the New York State Education Department and the New York City Board 
of Education had monitored the improvement efforts of these schools 
adequately. The audit included the examination of evidence recorded in 
accounting and operating records and the assessment of estimates, judgments, 
and decisions made by school management . Findings indicate that the Board and 
the Department did not always perform adequate analyses to determine the 
effectiveness of various consultant programs used by schools to improve 
performance on standardized tests. Improvement programs were not always 
effective, and some schools were able to improve their performance without 
the help of these costly programs. After completion of the audit, the New 
York Department of Education announced plans for a new information system to 
identify and track the academic performance of each student in the state. 

This report recommends that such a system contain the ability to correlate 
academic performance with improvement programs and resource allocations. 
Appendixes contain a list of major contributors to the report, the comments 
of state education department officials, and comments of officials from the 
New York City Board of Education. (SLD) 
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Dear Mr. Hayden and Mr. Levy: 

The following is our report addressing the actions taken by the State 
Education Department and the New York City Board of Education in 
monitoring improvement efforts of schools with low standardized test 
scores. 

This audit was performed pursuant to the State Comptroller’s authority as 
set forth in Article V, Section 1 of the State Constitution; Article II, 
Section 8 of the State Finance Law; and Article III, Sections 33 of the 
General Municipal Law. Major contributors to this report are listed in 
Appendix A. 
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Executive Summary 



New York State Education Department and New 
York City Board of Education 
Monitoring Improvement Efforts of Schools with 
Low Standardized Test Scores 



Scope of Audit 


Elementary and secondary schools in New York State are administered by 
more than 700 local school boards. In New York City, the schools are 
administered by the New York City Board of Education (Board). Local 
school boards are overseen by the State Education Department (Depart- 
ment), which is guided by the Board of Regents. Students in certain 
grades are required by the Board of Regents to take standardized tests to 
measure their competence in reading, writing and math. If a certain 
percentage of a school’s students (90 percent at the time of our audit) do 
not score high enough on a test to demonstrate a minimum level of 
competence in that skill, the school’s performance on that test is consid- 
ered to be below the State standard. The lowest performing of these 
schools may be classified by the Department as Schools Under Registration 
Review (SURR), and if their performance does not improve on subsequent 
tests, they may be required by the Department to close. 

During the 1996-97 school year, 1,521 of New York’s 4,111 schools 
experienced overall student test results that were below the State standard 
for one or more test. Between 1990 and 1997, a total of 159 schools were 
classified as SURR. As of December 1997, 54 of these schools had 
improved enough to no longer be classified as SURR, 7 schools had 
reorganized or closed, and 98 schools were still classified as SURR. Of 
these 98 schools, 94 were located in New York City. Our audit addressed 
the following question about the improvement efforts at SURR schools and 
other schools performing below the State standard during the period July 
1, 1995 through November 30, 1998: 

• Did the Department and the Board adequately monitor these 

improvement efforts? 


Audit Observations 
and Conclusions 


In recent years the Department and the Board have emphasized account- 
ability for school performance. This emphasis as well as the commitment 
to monitor and to recommend operational improvements to schools with 
academic performance which is below State standards is necessary and 
commendable. However, it is also critically important for the Department 
to be able to 'adequately determine which resource allocations and which 
program initiatives work most effectively and which work less effectively 



at iiiq)roving student academic performance in given school settings. With 
this type of performance measurement capability, the Department and the 
districts can share information and can make informed decisions which are 
most likely to maximize resource investment and produce desired academic 
performance outcomes. 

We found that the Board and the Department did not always perform 
adequate analysis to determine the effectiveness of various consultant 
programs being used by schools to improve performance on State 
standardized tests. We found that the improvement programs were not 
always effective and some schools were able to improve their performance 
without the help of these costly programs. Subsequent to the completion 
of our audit field work. Department officials advised of their plans to 
design a new information system that would identify and track the 
academic performance of each student on a statewide basis. We believe 
that it is important that such a system should be designed with functionality 
to correlate academic performance with improvement programs and 
resource allocations. (See pp. 5-9) 

We also concluded that the Department should monitor the performance 
trends at below-standard schools, determine which schools improved, 
identify reasons for improvement, and recommend that effective practices 
be adopted at other schools. (See pp. 11-12) 

The Department generally receives no State Aid appropriations which are 
specifically assignable to schools based upon academic performance that 
is below State standards. However, for the 1996-97 and 1997-98 school 
years, the Department reports that about $9 million of federal Title I 
Improvement Program funds and School Effectiveness Program funds were 
allocated to numerous projects that generally seek to improve performance 
of SURR schools and schools performing below State standard. While the 
Department had expenditure plans and various reports and analysis that 
recapped the completion and the accomplishment of certain of these 
projects, the Department did not maintain analysis showing how the 
various projects either individually or collectively directly affected the 
performance outcomes of students. While we recognize that such 
measurement is challenging and requires resources to implement, we 
maintain that such measurement is necessary to permit assessment of the 
relative effectiveness of various options. (See pp. 8-9) 



Comments of 
Department and 
board Officials 



Department officials generally agree with most of the recommendations in 
our report. Board officials do not specifically agree or disagree with the 
two recommendations in our report that pertain to the Board. Board 
officials believe that the audit does not fully appreciate the complexities of 
determining increases in students’ test scores. 
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Comments of New York City Board of Education Officials 



Introduction 



Background 



The Board of Regents (Regents) is responsible for setting education 
policies and for guiding, managing and monitoring the education system 
in New York State, which has the second largest elementary and secondary 
school system in the nation. In fulfilling these responsibilities, the Regents 
work with the Governor and Legislature, who also initiate education 
progr ams and ultimately control the State funds devoted to education. The 
16 Regents, who are elected to five-year terms by the Legislature, are 
headed by a Chancellor. 

The Regents are served by the State Education Department (Department), 
which administers the State’s education policies and programs. The 
Department oversees local school boards and monitors compliance with 
education laws and regulations. In addition, it offers technical assistance 
to the teachers and administrators at local schools. The Department 
employs about 3,300 staff, who are headed by a Commissioner who is 
appointed by the Regents. About 600 of the Department’s staff are 
involved in reviewing elementary and secondary education practices. 

While the Regents are responsible for setting education policy, the State 
education system is primarily a decentralized operation, rooted in the 
concept of local autonomy. Local school boards in about 700 school 
districts across the State make daily operating decisions for 4,111 
individual schools. Reflecting this decentralization, the Department’s 
1998-99 annual budget included about $295 million for State operations 
and more than $15 billion in State aid to localities. Some of this aid is 
distributed on the basis of a general formula that takes into account factors 
such as the student population in each school district. The remainder of 
the State aid is distributed to eligible school districts in accordance with 
specific education programs, including programs that provide remedial 
services and discourage students from leaving school prematurely. 

All public schools in New York City are overseen by the New York City 
Board of Education (Board). The Board employs more than 116,000 staff, 
including teachers, administrators, social workers, counselors and 
paraprofessionals to operate its 1,115 schools. While the high schools are 
managed directly by the Board, the elementary and intermediate schools 
are managed by 32 separate Community School Districts (CSD). The 
Board is supervised by the Regents, but various sections of the State 
Education Law assign many responsibilities specifically to the Board. 
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The Regents require that all students take certain standardized tests to 
measure their basic skills. For example, prior to the 1998-99 school year, 
students in the fifth grade were required to take Pupil Evaluation Program 
(PEP) exams in reading, writing and math. (Starting in the 1998-99 
school year, the PEP exams were replaced by more stringent exams 
correlating to the new learning standards, and given in the fourth and 
eighth grades.) In addition, in the eighth or ninth grade, students are 
required to take Preliminary Competency Tests (PCT) in reading and 
writing. If a student does not attain a certain score (called the State 
Reference Point) on a test, the student is determined to be making 
unsatisfactory progress in developing the basic skill measured by that test 
and is given remedial assistance in that subject area. Therefore, each test 
is intended to determine whether students have attained a minimum 
competence in a basic skill. 

Each year, the Department analyzes how the students at each school in the 
State performed on certain of these tests. If a certain percentage of a 
school’s students (90 percent at the time of our audit) do not score high 
enough on a test to demonstrate minim um competence in that skill, the 
school’s performance on that test is considered to be below the State 
standard. During the 1996-97 school year (the most recent school year for 
which data was available at the time of our audit), 1,521 of the 4,111 
schools in the State performed below the State standard on a test. 

The schools with low performance on a standardized test and having been 
identified as most in need of improvement may be classified by the 
Department as Schools Under Registration Review (SURR). Each SURR 
school is visited regularly and monitored by Department staff, who review 
the school’s operations and recommend operational improvements. If a 
SURR school does not improve its performance on the standardized test, 
its registration can be revoked by the Department. Since schools are not 
allowed to operate in New York State unless they are registered with the 
Department, such a school will be forced to close. SURR classifications 
are made by the Department, which is required by State regulation to 
consider the circumstances of each low performing school and classify as 
SURR only those schools that are most in need of improvement. 
Therefore, depending on the Com mi ssioner’s evaluation, some of the 
lowest performing schools may not be classified as SURR. 

Between the 1989-90 school year (when SURR classifications began) and 
the 1997-98 school year, a total of 159 schools were classified as SURR. 
By December 1997, 54 of these schools had improved enough to no longer 
be classified as SURR, seven schools had reorganized or closed, and 98 
schools were still classified as SURR. These 98 schools included 59 





elementary schools, 27 intermediary schools and 12 high schools; 94 of the 
98 schools were located in New York City. 


Audit Scope, 
Objective and 
Methodology 


We audited the actions taken by the Department and the Board in 
monitoring the improvement efforts of SURR schools and other schools 
performing below the State standard during the period July 1, 1995 
through November 30, 1998. The objective of our performance audit was 
to evaluate the adequacy of these monitoring efforts. To accomplish our 
objective, we reviewed and analyzed relevant Department and Board 
records, and we interviewed appropriate Department and Board managers 
and staff. We also interviewed educators at New York University and 
officials at the Educational Priorities Panel, a not-for-profit organization. 

We conducted our audit according to generally accepted government 
auditing standards. Such standards require that we plan and do our audit 
to adequately assess those procedures and operations included within the 
audit scope. Further, these standards require that we understand the 
Department’s and the Board’s internal control systems and their compliance 
with those laws, rules and regulations that are relevant to the operations 
included in our audit scope. An audit includes examining, on a test basis, 
evidence supporting transactions recorded in the accounting and operating 
records and applying such other auditing procedures as we consider 
necessary in the circumstances. An audit also includes assessing the 
estimates, judgments, and decisions made by management. We believe our 
audit provides a reasonable basis for our findings, conclusions and 
recommendations . 

We use a risk-based approach when selecting activities to be audited. This 
approach focuses our audit efforts on those operations identified through 
our preliminary survey as having the greatest probability for needing 
improvement. Consequently, by design, we use our finite audit resources 
to identify where and how improvements can be made. Thus, we devote 
little audit effort to reviewing operations that may be relatively efficient or 
effective. As a result, our audit reports are prepared on an “exception 
basis.” This report, therefore, highlights those areas needing improvement 
and does not address activities that may be functioning properly. 


Response of 
Department and 
Board Officials to 
Audit 


A draft copy of this report was provided to Department and Board officials 
for their review and comment. The comments of the Department and the 
Board were considered in preparing this report and are included as 
Appendix B and Appendix C, respectively. 



Within 90 days after final release of this report, as required by Section 170 
of the Executive Law, the Commissioner of the State Education Depart- 
ment shall report to the Governor, the State Comptroller, and the leaders 
of the Legislature and fiscal committees, advising what steps were taken 
to implement the recommendations contained herein, and where recommen- 
dations were not implemented, the reasons therefor. We also request that 
such a report be made by the Chancellor of the New York City Board of 
Education. 



Monitoring Resource Allocations and Improvement 
Programs 



In recent years the Department and the Board have emphasized account- 
ability for school performance. This emphasis as well as the commitment 
to monitor and to recommend operational improvements to schools 
performing below State standards is necessary and commendable. 
However, it is also critically important that the Department can adequately 
determine which resource allocations and which program initiatives work 
most effectively and which work less effectively at improving student 
academic performance in given school settings. With this type of 
performance measurement capability, the Department and the districts can 
share information and can make informed decisions which are most likely 
to maximize resource investment and produce desired academic perfor- 
mance outcomes. The following sections of this report discuss areas where 
such performance measurement capability can be applied. 



Improvement 

Programs 



We reviewed Department and Board procedures regarding their evaluation 
of the effectiveness of the different programs designed to improve students’ 
performance on the standardized tests. Board officials told us that several 
of the improvement programs used in New York City schools have been 
evaluated by consultants that were hired by the Board. We reviewed some 
of the reports prepared by these consultants and conclude that they are not 
adequate for evaluating the effectiveness of the improvement programs in 
City schools, because they generally are not based on systematic analysis 
to determine whether test scores improved at the schools using the 
programs. Moreover, in some instances, the effectiveness of a program 
was evaluated by the same consultant that developed the program. Such 
an evaluation may not be as reliable as an evaluation performed by an 
independent party. For example, the evaluations that the Board arranged 
for revealed: 

• Johns Hopkins University was contracted to review the implemen- 
tation of the Success For All program, which is used at a number 
of City schools. However, the University’s report only addressed 
a portion of these schools and did not include an analysis of the 
schools’ performance on the State standardized tests. 

• Metis Associates, Inc. was contracted to assess the implementation 
of the Success For All program at six elementary schools. 
However, the consultant did not determine whether the program 
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was successful at improving test scores. Rather, the consultant 
assessed the process of program implementation. 

A 

• Two other programs were evaluated by two other consultants 
(Lightspan Partnership and the McKenzie Group, Inc.). However, 
neither consultant analyzed the schools’ performance on the State 
standardized tests, and one consultant (the McKenzie Group) 
focused on whether program participants liked the program, not 
whether the program was effective. 

• Two consultants (Johns Hopkins University and Lightspan 
Partnership) evaluated programs that they had developed. 

Department officials told us they have reviewed some of the improvement 
programs used by SURR schools, but they have not reviewed all of these 
programs nor have they tracked the performance of all participating 
schools. The officials also told us that they have focused their efforts on 
systematic evaluations of the SURR program as a whole rather than on 
evaluations of individual improvement programs. For example, the 
Department produced an aimual report about the SURR program, hired 
New York University to perform an external evaluation of the SURR 
program, and asked schools to assess the Department’s registration review 
visits. We acknowledge the value of the Department’s efforts to evaluate 
the overall effectiveness of the SURR program. However, if the 
Department does not also evaluate the effectiveness of individual 
improvement programs, it may encourage schools to use, and provide 
funding for, programs that are not effective. 

Many of the SURR schools in New York City participated in the Models 
of Excellence program, which offered several different approaches to 
in^roving the performance of students and included a number of individual 
improvement programs such as the Success For All program. To 
determine whether the Models of Excellence program was effective, we 
reviewed the 1997 scores on the State standardized tests for third and sixth 
grade reading and math at 52 of the 79 New York City SURR schools that 
were participating in the program during the 1996-97 school year. We 
selected these 52 elementary and intermediary schools because they had 
participated in the program since 1993. 

We found that none of the schools with third grade test results performed 
at the State standard on the third grade reading test, none of the schools 
with sixth grade test results performed at the State standard on the sixth 
grade reading test, only 36 percent of the schools with third grade test 
results (17 of 47) performed at the State standard on the third grade math 
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test, and only 30 percent of the schools with sixth grade test results (8 of 
27) performed at the State standard on the sixth grade math test. Five 
schools did not have third grade test results and 25 schools did not have 
sixth grade test results because some of the schools did not include a third 
grade, many of the schools did not include a sixth grade, and a few 
schools did not have complete data for their test results. 

Therefore, most of the 52 schools performed below the State standard on 
these tests. However, it should also be noted that most of these schools 
improved their performance in 1997. Specifically, for 77 percent of the 
tests on which a school performed below the State standard in 1997, the 
school’s performance in 1997 was better than its performance in 1995. 
However, since the Department has not developed specific criteria for 
evaluating program effectiveness, we cannot conclusively evaluate the 
effectiveness of the program. We note that an earlier Department study 
of this program also identified conflicting indicators of program effective- 
ness, as the study found that most of the SURR schools participating in the 
progr am did not improve enough to be removed from the SURR list, but 
the percentage of participating schools that were removed from the SURR 
list was higher than the percentage of non-participating schools that were 
removed from the SURR list. 

(In response to our draft report. Department officials stated that the Model 
of Excellence program has been replaced by the Federal Comprehensive 
School Reform Demonstration Program, which is based on a similar 
concept of providing schools with the opportunity to replicate nationally 
recognized, research-based school improvement programs. The Depart- 
ment believes that inherent in this program are many of the evaluation 
components recommended by the audit.) 

To further evaluate the effectiveness of school improvement programs, we 
judgmentally selected six below-standard schools in New York City and 
five below-standard schools outside New York City at which sixth grade 
reading scores improved by at least 20 percent between 1995 and 1997. 
We asked officials at each of these schools whether their improvement 
could be attributed to any particular program. We found that three of the 
City schools and two of the non-City schools used improvement programs, 
while the other three City schools and the other two non-City schools did 
not use improvement programs. Moreover, we were consistently told that 
the improvement in reading was the result of the staff s extra effort rather 
than any special program. Therefore, some schools may be able to 
improve their performance without the help of costly prepackaged 
improvement programs. 







7 





The Department and the Board must work together to determine which 
approach is best for each school, and if a prepackaged program is the best 
approach for a particular school, the Department and the Board need to be 
able to help that school select the program that is best suited for that 
school. If the Department and the Board are to be able to provide such 
guidance, they must evaluate the effectiveness of the available programs. 

Board officials responded that it is difficult to isolate the effect of a single 
program from the effects of other programs in a school. We acknowledge 
the difficulty of this task. However, unless individual improvement 
programs are evaluated and the more effective programs identified, school 
officials may not have a reliable basis for selecting an approach to helping 
their students become competent in basic skills. 

(Subsequent to the completion of field work for our audit. Department 
officials advised us of their plans to implement a new statewide student 
information system which would be the foundation for tracking and 
measuring the academic performance of the State’s schools districts and 
students. We believe that the planning for this system should also include 
the capability to correlate resource allocations and improvement programs 
to actual academic outcomes. It would also appear that this system would 
not only facilitate efforts at improving results of schools below State 
standards, but would also help to further improve the results of schools 
performing at or above State standards.) 


Resource 

Allocations 


The Department generally receives no State Aid appropriations which are 
specifically assignable to schools based upon academic performance that 
is below State standards. The Department did, however, administer one 
legislative member item of $2 million for the 1997-98 year that was 
appropriated to fund initiatives intended to help improve SURR school 
performance. In addition, the Department does annually administer the 
Federal Title I Improvement Program and the State School Effectiveness 
Program which are targeted to economically disadvantaged districts and are 
also intended to improve educational quality and student academic 
performance. For the 1996-97 through 1997-98 school years, the 
Department reports that about $9 million of funds for these two programs 
have been used for numerous • projects that generally seek to improve 
performance of SURR schools and schools performing below State 
standard. The Department maintained records supporting the planning for 
the uses of these funds and we were able to trace the use of these funds 
to specific expenditures and related projects. In addition, the Department 
provided various reports and analysis that recapped the completion and 
accomplishment of certain of the projects. However, the Department did 
not maintain measurements of how the various projects either individually 



or collectively affected the performance outcomes of the students in SURR 
schools or other schools performing below State standard. We recognize, 
as the Department pointed out to us, that constructing such a metric would 
be very challenging and would require reallocation of scarce resources. 
Nevertheless, as previously discussed, if the Department is to facilitate 
overall improvements in student and school academic performance, it is 
critically important that the Department is able to assess the relative 
effectiveness of its investments of resource allocations. If justifiable 
resource limitations hinder this assessment, the Department can bring 
further attention to this through the State budget process. 



Recommendations 

To the Department: 

1 . Develop the capability to adequately determine which resource 
allocations and which program initiatives work most effectively 
and which work less effectively at improving academic 
performance in given school settings. 

(Department officials agree with recommendation number 1. 
They indicate that they are in the process of identifying an 
organization that will review the process of allocating resources 
and their impact on improvement of academic performance in a 
given school setting.) 

To the Department and the Board: 

2. Ensure that all school improvement programs are evaluated 
independently to determine the extent to which they are effective 
at improving student scores on State standardized tests. 

3. Determine why some school improvement programs are more 
effective than others, and determine whether certain types of 
programs are likely to be more effective in certain circumstances. 




15 - 



9 



Recommendations (Cont.’d) 



(Concerning recommendation number 2, Department officials 
agree with the notion that consultant programs should be 
evaluated for effectiveness. However, officials state that the 
Department should not favor or appear to favor any fee based 
programs available to schools and districts monitored by the 
State. They also state that the role of the Department should be 
to assist districts and schools in making good choices. 
Department officials agree with recommendation number 3. They 
add that an evaluation is in process for the Comprehensive School 
Reform Demonstration program.) 

(Board officials do not specifically agree or disagree either with 
recommendation number 2 or recommendation number 3. They 
state that we have not fully appreciated the complexities of 
deterriiining increases in students’ test scores and, therefore, are 
attempting to reduce multi-variable relationships to a single causal 
explanation. They add that program results are apparent over 
time and care must be exercised to prevent premature judgement.) 

Auditors’ Comments : We do not dispute that the Department may 
be able accomplish the intent of recommendation number 2 
through effective guidance to districts and schools. With respect 
to Board comments, we reiterate that our report already 
acknowledges that it is difficult to isolate the effects of a single 
program from the effects of other programs in a school. 
However, we believe the Board must respond to this challenge so 
that those programs that truly have benefit in terms of increasing 
student performance can be recognized and promoted. In this 
regard, the guidance being contemplated by the Department may 
also be of use to the Board. 
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Monitoring Below-Standard Schools 

As is shown in the following table, as of December 1997, a total of 1,521 
schools were performing below the State standard and 98 of these schools 
were classified as SURR: 



Location 


SURR 


Below Standard 


New York City 


94 


894 


Buffalo, Rochester, Syracuse or Yonkers 


1 


151 


Small cities 


0 


45 


Suburban areas 


1 


41 


Rural areas 


2 


390 


Totals 


98 


1,521 



Therefore, significant improvements are needed in student performance at 
these schools. However, while SURR schools receive extra attention from 
the Department to help them improve their performance, the remaining 
below-standard schools do not receive this same attention. Moreover, if 
the performance of the students at these below-standard schools does not 
improve, the schools may be reclassified as SURR. For example, in 
November 1997, 20 below-standard schools in New York City were 
reclassified as SURR, while 18 New York City schools were removed 
from the SURR list. 

We believe more of an effort should be made to improve the performance 
of students at the below-standard schools not classified as SURR in order 
to prevent them from becoming SURR. In particular, the Department 
could monitor performance trends at below-standard schools, determine 
which schools were improving, identify the reasons for the improvement, 
and recommend that the effective practices be adopted at other schools. 

Department officials responded that, if they were to commit additional 
resources to monitor below-standard schools, they would have to decrease 
the resources dedicated to SURR schools, and would consequently 
jeopardize their ability to improve the performance of SURR schools. We 
acknowledge the difficulties involved in monitoring additional schools. 
However, the Department is not limited to the resources dedicated to 
SURR schools, and evaluations may need to be made about whether 
resources can be shifted from other areas without affecting the SURR 



program. We note that, unless student performance can be improved at 
the below-standard schools not classified as SURR, a significant number 
of students will continue to lack basic skills. 



Recommendation 

To the Department: 

4. Monitor performance trends at below-standard schools, determine 
which schools are improving, identify the reasons for the 
improvement, and recommend that the effective practices be 
adopted at the below-standard schools where performance is not 
improving. 

(Department officials agree with recommendation number 4. 
They indicate that they have drafted regulatory amendments to 
identify schools performing below accountability criterion, to 
establish yearly progress targets for schools, to require 
improvement plans when adequate progress is not made, and to 
recognize schools making rapid improvements.) 



Classifying Schools as SURR 



The schools with the lowest performance on a standardized test may be 
classified by the Department as SURR, depending on the evaluation of the 
circumstances affecting each school’s performance. If a school is classified 
as SURR, it receives additional assistance from the Department. The 
Department uses cutoff points to identify potential SURR schools. For 
example, in the 1995-96 school year, if fewer than 30 percent of a 
school’s third grade students achieved the State Reference Point on the 
reading test, the school was considered for SURR classification. For the 
1995-96 and 1996-97 school years, we reviewed the process used by the 
Department to identify potential SURR schools and found that certain 
improvements can be made in this process, as is shown by the following 
examples from a total of 135 different schools: 

• A total of 25 low performing schools were not considered for 
SURR classification, even though their performance exceeded the 
Department’s cutoff points, because the performance data submitted 
by the schools was considered inaccurate by the Department. 
Department officials told us they were informed by district officials 
that the data was inaccurate and, on the basis of previous submis- 
sions and discussions with district officials. Department officials 
accepted that the data was inaccurate and allowed revised data to 
be submitted. However, Department officials did not verify that 
the original data was in fact inaccurate. We believe the Depart- 
ment should verify the data in such instances. Department officials 
told us that, beginning with the 1997-98 school year, the district 
superintendent is required to certify that test data is correct before 
it is submitted to the Department and any revised data must be 
supported by appropriate documentation. 

• A total of 84 low performing schools were not considered for 
SURR classification, even though their performance exceeded the 
Department’s cutoff points, because they were alternative schools 
intended to educate at-risk students who transfer from traditional 
high schools. During the two years we examined, it was the 
Department’s policy to exclude such schools from the SURR list. 
However, we believe such schools would benefit from the 
improvement efforts that are made at SURR schools. Beginning 
with the 1997-98 school year, the Department revised its policy 
and no longer automatically excludes alternative high schools from 
SURR classification. 



• A total of 10 low performing schools were not considered for 
SURR classification, even though their performance exceeded the 
Department’s cutoff points, because fewer than 20 students took 
the standardized test used in making the SURR classification. It 
is the Department’s policy to exclude such schools from the SURR 
list because Department officials believe the results of such small 
samples are not reliable indications of overall student performance 
at a school. While we understand this policy, we would also 
suggest that an appropriate method should be developed for 
evaluating the performance of schools with small test populations. 



Recommendations 

To the Department: 

5. Ensure that any revised performance data submitted by a school 
is either verified or appropriately documented. 

6. Develop a method for evaluating the performance of schools with 
small test populations. 

(Department officials agree with recommendation number 5 and 
with recommendation number 6. They state that, subsequent to 
the time period covered in our audit, the Department revised its 
process for data verification and conducted spot audits of data. 
Officials add that the Regents adopted amended regulations that 
put in place a method using data from multiple years to evaluate 
schools with small test populations.) 
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Appendix A 



THE STATE EDUCATION DEPARTMENT / THE UNIVERSITY OF THE STATE OF NEW YORK / ALBANY, NY 12234 




Chief Operating Officer 
Tel. (518)474-2547 
Fax (518)473-2827 
E-mail: rcate@mail.nysed.gov 



March 31, 2000 

Mr. Jerry Barber 
Audit Director 

Office of the State Comptroller 
A.E. Smith State Office Building 
Albany, NY 12236 

Dear Mr. Barber: 

I am responding to the draft audit report (98-S-34) addressing the State 
Education Department’s and the New York City Board of Education’s monitoring of 
improvement efforts of schools with low standardized test scores. 

Recommendation 1: Develop the capability to adequately determine which resource 
allocations and which program initiatives work most effectively and which work less 
effectively at improving academic performance in given school settings. 

We agree with the recommendation . We are in the process of identifying an 
organization that will be assigned the task of reviewing: 

□ the process of allocating resources; and 

□ their impact on the improvement of academic performance in given school 
settings. 

Recommendation 2: Ensure that all school improvement programs are evaluated 
independently to determine the extent to which they are effective at improving student 
scores on State standardized tests. 

We agree partially with the recommendation . This question requires a two-part 
response. First, the New York State Education Department (NYSED) concurs with the 
notion that "consultant programs" should be evaluated for effectiveness. However, we 
do not believe it appropriate that NYSED favor or appear to favor any fee based 
program that is available to schools and districts monitored by the State. The decision 
to use or not use such programs must rest with the local education agency and school. 
Each local district in turn should evaluate how successful the replication of a particular 
program has been In the district. 
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The role of NYSED should be to assist districts and schools in making good choices. 
With this in mind, one of the NYSED collaborators in the Comprehensive School 
Reform Demonstration (CSRD) program initiative (Teacher Centers Consortium of the 
United Federation of Teachers) commissioned the LAB at Brown University to conduct 
a CSRD model alignment study. The purpose of this study is to determine the degree 
of match between each of several CSRD models being implemented in New York State 
and the New York State Learning Standards. A multi-chapter volume with related staff 
development materials will be ready for dissemination shortly. 

Recommendation 3: Determine why some school improvement programs are more 
effective than others, and determine whether certain types of programs are likely to be 
more effective in certain circumstances. 

We agree with the recommendation . In the case of districts and schools involved in the 
CSRD program, an evaluation is in process (see response to recommendation # 2 
above). It is expected that the data derived will provide information concerning how the 
implementation of an externally developed model may Impact a comprehensive school 
improvement program. 

Recommendation 4: Monitor performance trends at below-standards schools, 
determine which schools are improving, identify the reasons for the improvement, and 
recommend that the effective practices be adopted at the below-standards schools 
where performance is not improving. 

We agree with the recommendation . The Department has drafted amendments to 
Commissioner's Regulations that would: identify schools that are performing below a 
school accountability criterion, establish adequate yearly progress targets for such 
schools, and require schools to develop school improvement plans when adequate 
progress is not made. Regulations will also authorize the Commissioner to recognize 
schools that make rapid improvement. We are in the process of identifying an 
organization that will develop a process to identify the pattem(s) of practices used in 
schools that have been described as below standard but have shown significant 
improvement. 

Recommendation 5: Ensure that any revised performance data submitted by a school 
is either verified or appropriately documented. 

We agree with this recommendation . Subsequent to the time period covered by the 
audit, July 1, 1995 to November 30, 1998, the Department revised its process for 
verification of submitted data and conducted spot audits to verify data reported. The 
Department's planned implementation of a statewide student record system, as 
described in the audit, will contribute to the Department’s ability to verify data. 



B-2 

o 

ERIC 



23 



3 



Recommendation 6: Develop a method for evaluating the performance of schools with 
small test populations. 

We agree with this recommendation. In October 1998, the Regents adopted amended 
Commissioner’s Regulations that put in place a method using data from multiple years 
to evaluate schools with small test populations. 

Other Comments: 

Department staff has reviewed the audit report narrative and identified the 
following inaccuracies: 

□ Page 1 . paragraph 1 : Members of the Board of Regents are appointed to five- 
year terms, not seven-year terms as stated in the report. 

□ Page 2, paragraph 1 : The PEP writing test was administered in Grade 5, not in 
Grades 3 and 6. 

• □ The Department requests that the audit clarify that the Models of Excellence 
program referred to in the draft audit is ho longer in existence. Many Schools 
Under Registration Review (SURR) now participate in the Comprehensive 
School Reform Demonstration Program (CSRD). CSRD makes funds available 
to encourage schools, especially Title I schools, to work towards comprehensive 
school reform by creating a comprehensive school reform program that includes 
a' sound, research-based, established school reform model. The main objective 
of the program is to use these programs to enable all children to meet 
challenging statewide standards. SED believes that inherent in the CSRD model 
are many of the evaluation components recommended by the audit. 

The Department further requests that the text more clearly identify those 
statements of fact that pertain to the New York City Board of Education and those that 
pertain to the State Education Department. For example the bullets on page 5 of the 
draft refer to evaluations done by the New York City Board of Education rather than the 
State Education Department. However, it is difficult to determine this fact from the text. 

If you have any questions, please contact Shelia Evans-Tranumn or Ira Schwartz 
at (718) 722-2796. 



Sincerely, y y' ^ ^ 




Richard H. Cate 



cc: James A. Kadamus 

Shelia Evans-Tranumn 
Ira Schwartz 



* State Comptroller's Note 

1 . Certain matters addressed in the draft report were revised or deleted in the final report. Therefore, 
some agency comments included in Appendix B may relate to matters no longer contained in this 
report. 
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Board of Education of the City of New York 

HAROLD O. LEVY, Chancellor 



J 10 LIVINGSTON STREET • BROOKLYN. NY 11201 



April 3, 2000 



Offic** 



Mr. Jeny Barber, Audit Director 
Office of the State Comptroller 
A. E. Smith Building 
Swan Street 
Albany, NY 12236 



RcOt 



MAILROOM 



Re: Draft Audit Report on SED and BOE’s Monitoring 
Improvement Efforts of Schools with Low Standardized 
Test Scores (98-8-S-34) 



Dear Mr. Barber: 

Enclosed are the Board of Education’s responses to the draft audit report. We are 
pleased that the New York State Comptroller’s office acknowledges the Board’s 
emphasis on accountability for school performance. 

Although only two of the six audit recommendations are addressed to the Board, 
we believe that the auditors have not fully appreciated the complexities of determining 
increases in students’ test scores and, therefore, are attempting to reduce multi- variable 
relationships to a single causal explanation. Additionally, the results of program 
implementation are apparent over a period of time, and care must be exercised to prevent 
premature judgement. 

As we did during the audit, we again share with you documents that will aid in 
understanding our approach. 

The Board of Education always seeks improvement in student performance and 
continually reviews and assesses the effectiveness of programs. 



Sincerely, 




Oudith Rizzo, Ed.D 

Deputy Chancellor for Instruction 



JAR:bym 

Enc. 

C: William Casey 



Robert Tobias 
Edward Torres 



Betty D. Arce 
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MEMORANDUM 



TO: MICHAEL KING 

FROM: William P. Casey 

SUBJECT: Response to NYSC’s Audit of BOE and SED’s Procedures for Low Performing 

Schools Program 

Although most of the recommendations contained in the draft audit report are 
focused on the Stale Education Department, the allegations made relating to the Board’s conduct 
appear to be as follows: 

a. Analyses of intervention programs are not adequate to determine their 
effectiveness. 

b. Some schools are able to improve their performance without costly 
programs. 

c. Evaluation consultants focused on evaluation of program 
implementation rather than on the effect of the program on test scores. 

d. In some instances, programs were evaluated by the same consultants 
that developed the program. 

Reponses to the above concerns are: 

a. Although it is tempting to suggest that increases in students’ test scores can be traced 
to a single causal intervention, this is never the case. In a two-variable relationship, there is 
obvious cause and effect. However, educational innovations include several features, each of 
which becomes an intervening variable. As was indicated in a previous response, all 
educational innovations must be evaluated based on a number of factors, including, but not 
limited to: the size and scope of the innovation; the quality of administrative leadership; a school 
culture for change; teacher leadership and support for the innovation; emd the quantity and 
quality of continuous professional development. Any one of these factors can and does impact 
on the ability of an educational innovation to raise student achievement. 

b. It is true that some schools are able to improve student performance without the need 
to adopt ‘"packaged” programs. They are able to succeed because they have developed their own 
school-based program that includes the research-based practices and procedures commonly 
found in many of the programs adopted by schools. The ability to accomplish this relies on 
many factors, including those mentioned above (i.e. supportive administrative leadership, teacher 
leadership and support, a school culture for continuous change, on-going professional 
development, and adequate resources). Also, since many of these factors rely on the ability of 
individuals to inspire, motivate, and persevere, they are clearly not able to be quantified. 
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c. Additionally, it may require a longer time to measure the effect of any one program 
on the achievement of the students. Educators have a history of making premature judgements 
about the effectiveness of educational innovations and discarding them before they have had an 
opportunity to demonstrate effectiveness. Analyses of educational programs require the 
examination of their implementation, in the initial stages of program implementation, it is 
important that the research-based practices, upon which the program was developed, are being 
executed in the way in which they were designed. A modification in any aspect of performance 
can dramatically effect the outcome of the entire program. Therefore, it is appropriate that a 
program evaluation should focus on the implementation of these practices to provide valuable 
assessment data for on-going program monitoring. 

c. See attached response from Robert Tobias. 



WPC:sk 



c: Judith A. Rizzo 



Betty Arce 
Len'L. Davis 
Sandra Kase 
Robert Tobias 
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Board of Education of the City of New York 

HAROLD O. LEVY, e^^4McdU>>t . 

DIVISION OF ASSESSMENT AND ACCOUNTABILITY • 1 10 LIVINGSTON STREET. ROOM 728. BROOKLYN. NY 1 1201 



ROBERT TOBIAS 
EXECUTIVE DIRECTOR 
TELEniONE (71B) M5-3767 
FAX (7I8>US-S2«8 



U>R1 MEI 

DETUTY EXECUTIVE DIRECTOR 
telephone (7II)«3«T77 
FAX (711) 935-526S 



MEMORANDUM 



TO: 

FROM: 

SUBJECT: 



March 6, 2000 

MICHAEL KING 
Robert Tobias 

Review of Draft'state Comptroller’s Audit Report 98-S-34 
“Monitoring Improvement Efforts of Schools with Low Standardized 
Test Scores" 




We have reviewed the draft copy of the State Comptroller’s audit 
report “Monitoring Improvement Efforts of Spools with Low Standardized Test 
Scores" (Report #98-S-34). Most of the audit’s conclusions and recommend- 
ations are directed at the State Education Department and their oversight of 
programs put into place in Schools Under Registration Review (SURR) to 
improve students’ academic performance. 

Cniy two of the audit’s recommendations are addressed to the 
Board (as well as to the State Education Department). These recommend that 
all school improvement programs be evaluated independently to ascertain their 
impact on students’ performance on state tests, and that the relative 
effectiveness of different programs be determined. As noted in the report, it is 
difficult to isolate the effect of a single program from the effects of other programs 
in the school. 



Further, while it is true that the designers of some of the models 
have conducted evaluations of their own programs. New York City has not used 
the data from these evaluations in either the selection of the models or in the 
evaluation of their efficacy. Within the constraints to definitive evaluation 
mentioned above. New York City has attempted to evaluate the efficacy of these 
programs, most notably in the attached internal report titled, “Schools Under 
Registration Review/Models of Excellence 1994-95.” 

In addition and perhaps more fundamentally, several of the school 
improvement models examined in the audit are designed to change the culture of 
belief systems and practice in schools over time. In many of these models, 
improvement in student performance is a long-term objective that is believed to 
occur as a result of changes in culture and practice. This long-term change 
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March 6, 2000 



model requires study overtime, first to assess its qualitative impact on school 
culture and practice, and later its effect on student outcomes. 

New York City has designed such a two-phase evaluative process. 
The Performance Assessment in Schools Systemwide (PASS) process 
qualitatively examines changes in school culture and practice by assessing 
schools against the standards and practice of exemplary schools. As indicated 
in the attached PASS Review Guide, schools are judged on 77 performance 
indicators relating to the essential elements of an exemplary school in terms of 
the extent to which they are “Meeting Standards," “Approaching Standards," or 
are “Below Standard" on each element. In the second phase of this process, 
changes in student performance are examined using standardized test scores. 
Since most of the models of excellence have only recently been implemented, in 
many cases they have manifested their impact in PASS reviews, and are only 
beginning to show measurable gains on student outcomes. 



RT/agh 

Att. 
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